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Abstract: Planar Fourier capture arrays (PFCAs) are optical sensors built entirely in standard mi- 
crochip manufacturing flows. PFCAs are composed of ensembles of angle sensitive pixels (ASPs) 
that each report a single coefficient of the Fourier transform of the far-away scene. Here we charac- 
terize the performance of PFCAs under the following three non-optimal conditions. First, we show 
that PFCAs can operate while sensing light of a wavelength other than the design point. Second, if 
only a randomly- selected subset of 10% of the ASPs are functional, we can nonetheless reconstruct 
the entire far-away scene using compressed sensing. Third, if the wavelength of the imaged light is 
unknown, it can be inferred by demanding self-consistency of the outputs. 
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1. Introduction 

1.1 Planar Fourier Capture Arrays 

Planar Fourier capture arrays (PFCAs) are ensembles of angle sensitive pixels (ASPs) [0, ED that 
directly capture the Fourier transform of a far-away image. The light sensitivity of an individual 
ASP is a sinusoidal function of incident angle ED. A carefully-chosen ensemble of ASPs can 
report the entire Fourier transform of the far-away scene up to the Nyquist limit set by the highest- 
frequency ASP in the ensemble [HD- As ASPs are extremely thin, light and cheap to manufacture, 
PFCAs have the potential to become a disruptive technology in the fields of sensing and robotics 
EL 0, 0, [ZD when volume, mass or cost constraints are paramount. 

For PFCAs to reach their full potential, it would be useful to understand some of the limita- 
tions and freedoms afforded by this new class of imager. Our first PFCA was optimized for green 
incident light, and the reconstruction algorithms we used previously O assume all sensors are 
functional. In this publication we thoroughly report PFCA design considerations, then investigate 
the performance of PFCAs imaging light of a different colour than the design specification, both 
when that colour is known a priori and when it must be determined blindly. Further, we will show 
that by using a signal-processing technique called compressed sensing, we can reconstruct images 
even when the outputs of only a small fraction of the ASPs are available. 
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Figure 1. Interaction Between Incident Light and ASPs. Left: light incident at 10° produces spatial maxima 
that align with gaps in the second grating, resulting in a large photocurrent. Center: light incident at 25 ° 
produces spatial maxima that are largely blocked by the second grating, resulting in a small photocurrent. 
Right: photocurrent is a sinusoidal function of incident angle. 



Talbot Effect 




Figure 2. The Talbot Effect. Wave simulation of monochromatic light striking a grating at the normal. The 
grating periodicity is recapitulated at regularly- spaced depths below the grating; a phenomenon known as 
the Talbot effect. The Talbot depth is shown at right. 

1.2 Angle Sensitive Pixels 

Angle sensitive pixels (ASPs) [0, ED are photosensitive pixels that, using diffraction gratings, couple 
to far-field light sources with an efficiency that is a sinusoidal function of incident angle. ASPs 
attain their angle sensitivity through the moire effect using two metal gratings with identical spacing 
but at different heights above a photodiode below (see Figure |T|). Due to the vertical displacement of 
the gratings, the angle subtended by one spatial period of the top grating viewed by the photodiode 
is slightly less than that subtended by the bottom grating. This angular disparity invokes the moire 
effect, making the effective transmission aperture a sinusoidal function of incident angle. Both of 
the gratings can be manufactured using metal interconnect layers intrinsic to the CMOS process, 
and the photodiodes can be manufactured using intrinsic semiconductors, meaning ASPs can be 
manufactured entirely using existing unmodified CMOS process flows. 

The moire effect is essentially a ray optics effect, however at the small spatial scale of CMOS 
structures photons cannot be modeled as non-diffracting rays. Fortunately, the moire effect can 
be rescued by exploiting the Talbot effect flEL 0]. The Talbot effect is the property that a regular 
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periodic grating illuminated by monochromatic light creates self-images at integer multiples of the 
Talbot depth Zj = ^ — | (see Section 1.3) where p is the grating pitch (spatial period) and X is 



the wavelength of light (see Figures |T| and 0). By ensuring that the second grating is located at an 
integer multiple of half 1 the Talbot depth, the moire effect is rescued and further diffraction effects 
can be neglected. 

The transfer function of light incident on an ASP can be modeled as in Equation [TTT|. 

fl = / (l-mcos(&e + a))F(e)(l + Tj), (1.1) 

where R is the readout of the ASP, Iq is proportional to the light flux at the ASP, 6 is the incident 
angle along the sensitive axis, b is the angular sensitivity (designed to range from 7 to 39 - see 
Equation [ZT]), m is the modulation depth of the ASP (see Equation P^|) , a is a designable phase 
offset caused by a displacement between the top and bottom gratings, F(6) is a slowly-varying 
aperture function and 7] is multiplicative noise. 

1.3 Deriving the Talbot Depth 

The | term of Z? = ^ — | is often omitted especially in considerations of Talbot periodicity rather 
than absolute depth and when p 2 A 2 , but is derivable from a consideration of optical path lengths 
and provides a noticeable increase in accuracy when modeling and designing ASPs. Consider the 
geometry of Figure ^|. For a local minimum to occur at r\ = the path length difference between 



f and r 2 = J (f 1 ) + p 2 should be §. Thus, 
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2. Designing PFCAs 

An ASP's angular sensitivity (Z? from Equation [ET1) can be found by considering the angular dis- 
placement required for a lobe of the Talbot diffraction pattern to traverse a full period of the ana- 

'At half-integer multiples of the Talbot depth the periodicity of the initial grating reemerges, but with its phase 
reversed; see Figures [I] and 
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Figure 3. Calculating the Talbot Depth. The geometry in this configuration leads to our expression for the 
half Talbot depth in Equation [Oj 



lyzer grating, as in Equation [2TT1. 



b = 2 k 



Zeffectr 

pn 



(2.1) 



where Zeffective is the effective vertical displacement between gratings (see Equation and n is 
the refractive index of the medium, in this case S1O2. The depths z at which modulation depth m is 
maximal correspond to integer multiples of half the Talbot depth, as in Equation g^ . 



m maximal when z — a 
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2.1 Effective Depth 



Since the PFCA was built, detailed measurements have indicated that when using Equation IZTj b is 
more accurately predicted by replacing the vertical separation between gratings z with an effective 
depth, Zeffective* set by the nearest half-Talbot depth as follows. 



^closest Talbot — 



Zeffective — 




(2.3) 



At optimal depths for a particular A, Zeffective = z. However, for non-optimal As, Zeffective and 
thus b vary as a function of A, an effect previously observed [IE EL ED but not explained. 

The phenomenon that the effective depth is the closest ideal Talbot pattern was demonstrated in 
a test structure accompanying our original PFCA prototype. A long, linear array of photodetectors 
was built with smoothly varying pitch. The observed angular sensitivity b as a function of pitch 
p is plotted in Figure f|, along with predictions based upon actual and effective depths applied 
to Equation |ZT] . It can be seen that parameter b is better predicted by the effective depth from 
Equation ^3] than the actual separation of the gratings. 

2.2 Combining ASPs into a Fourier-complete array using CMOS 

The metal interconnect layers of a CMOS process are manufactured at process-specific heights 
above the silicon substrate; to work within an established CMOS process it is necessary to manu- 
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Figure 4. Closest Talbot Pattern Determines b. Observed b(p) agrees better with the assumption that 
^effective equals the closest optimal Talbot depth, and not the manufactured depth. Blue circles: measured b 
as a function of pitch for incident green light on a series of devices spanning a range of p from 0.7 to 1.3 
microns. Red xs: modeled b assuming Equations ^?T| and [231. Black line: modeled b assuming ^effective = z 
and Equation \H\. Grey bars: depth of closest Talbot pattern. 
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Figure 5. Selecting Devices for the PFCA. Filled black circles indicate manufacturable devices with maxi- 
mal m for 520nm light; open circles indicate p and (A -dependent) b for the suite of devices we manufactured. 



facture all metals at one of the specified depths. There are therefore a discrete spectrum of inter- 
metal depths available. Using Equations an d it is possible to determine the spectrum of 
manufacturable b values that have locally-maximal m (see filled circles of Figure |5|). 

For Fourier completeness, it is important to sample Fourier space with an adequate density 
to ensure coverage. The more densely-packed the Fourier frequency measurements, the larger 
the angular region that can be observed with Nyquist sampling. As reported previously flU], the 
relationship between the maximum allowable incident light angle h and the maximum difference 
in b between consecutive designs is 

180° 

h=-^. (2.4) 
V2Ab 

The first prototype we built included devices with p tuned for a range of wavelengths other than 
the design point (520 nm) by manufacturing devices with p differing slightly from that of locally- 
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optimal devices (see Equation \U2\). The ASPs included in the manufactured device are shown as 
empty circles in Figure |5|; note each design of constant p and z has different b for different As. 
We arranged 1444 ASPs in two PFCAs with complementary as (see Equation |Q|) . All ASPs of a 
given design (combination of z and b) are found in one of 18 concentric rings arranged around four 
low-Z? devices described elsewhere IfTTHl. Rings with higher b are placed further from the center 
to allow for a greater variety of grating orientations, as is required for Fourier tiling. Although in 
general higher b requires lower p, as seen in Figure |5] the first 6 devices chosen have the opposite 
trend since they are all designed to the first half-Talbot depth for four different metal layers. The 
schematics in Figures |5| and show p increasing as one moves outward from the center, but in 
general this is not the case and the outermost device rings have the smallest p and highest b. 

Resolution limits are set by the maximum b of any device b max . This Nyquist limit corresponds 
to two rows or columns of pixels per period of the highest-^ device. For h « 45 °, the total number 

of effective pixels is (^^j , or approximately 400 pixels for this prototype 2 . 

The prototype's bs span the space relatively well for blue and green light (largest Ab is 3.6 and 
3.3 respectively; corresponding hs are 35° and 38°), but a gap in bs for red light in the b = 13-21 
range reduces our expectations that red images will be recovered well for half-angles h > 17°. 
Furthermore, as the pitch of a device approaches a single wavelength, its signal-to-noise ratios 
degrades, such that red light will intrinsically provide worse performance in fine-pitch ASPs. Both 
of these disadvantages working under red light contribute to poor reconstructions (see Section ||). 
The manufactured device is seen in Figure 

2.3 Reconstructing Images 

The prototype PFCA was presented with calibration (Figure [TjA) and test images using a square 
CRT area 20cm on a side, 22.86cm from the PFCA 3 for an h of 31.7° at the square's corners. This 
h is small enough to allow full Fourier coverage for blue and green, but not red light (see Equation 

To reconstruct images computationally, we performed the following operations. First, we com- 
puted the relationship between the photocurrent and the observed voltage drops at the photodiodes 
by fitting a quadratic function to V(t) under constant illumination from an LED lit by a steady 
power source. Then we threw out all data from any ASP that was saturated for more than 30% of 
all image presentations (24.5% of all sensors). This was necessary because long range diffusion of 
photo-generated carriers tended to add a background photocurrent to many of the ASPs close to the 
edge of the array. Next, we performed reverse correlation f ITTIl between inferred photocurrents and 
the calibration images with the following steps. We computed the pseudoinverse of the matrix 4 of 
calibration stimuli C, and multiplied it by the matrix of responses R to obtain an estimate of the 
individual transfer functions H as follows. We assume R is linearly related to C by H, therefore 

R = CH (2.5) 

2 As ASPs' transfer functions are not pure Fourier components but contain some harmonics, a limited amount of 
information outside the expected Nyquist limit is available so long as the coverage of Fourier space is overcomplete and 
noise levels are low. 

3 As 23cm ^> 570/im (the PFCA's size), images presented are in the far field regime where the light field at each 
point in the PFCA is essentially identical. 

4 Matrix quantities such as C shall be in capital bold, vector quantities in lowercase bold. 
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Figure 6. Manufactured PFCA. Concentric rings of ASPs with increasingly higher b yield a complete 
Fourier description of the light intensity from the far field. Slowly varying orientation is evident from the 
enlarged section, where schematics show different metal layers in different colours. 



A B C 




Figure 7. PFCA Calibration. Transfer functions of each pixel are found by presenting 6710 random binary 
50 x 50 calibration images (A) on a CRT screen (presentation time 16.7 ms each) to the array (B) and 
performing reverse correlation between the observed photocurrent of each sensor and the image presented. 
The kernels of three ASPs are shown (C); these resemble Fourier components. 

C is not square since there are 6710 patterns of 2500 free parameters each, so we can compute 
and multiply both sides of Equation |23| by C's pseudoinverse (C T C) _1 C T . 

(C^-^R = (C T C) _1 (C T C)H. 

(^C) -1 ^ = H. (2.6) 

Having computed H (sample rows of H are shown in Figure 0C), we need to find a way to 
invert it so we can reconstruct a new stimulus s new given a new set of responses r new . We computed 
the eigenspectrum of HH T (see Figure |8|) and found a regularized pseudoinverse of H using ridge 
regression lOl as follows. Ridge regression de-emphasizes eigenvectors of HH T that have little 
power. It is an acknowledgement that some stimulus eigenvalues are not represented strongly in 
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R and thus should be attenuated to avoid letting noise in poorly-determined stimulus components 
overwhelm signal in well-determined components. 5 




'"new — S new H 

r„ewH T (HH T + aiy 1 w s new (2.7) 

Here a was chosen by inspection of the eigenspectrum of HH T , and values of 30, 20 and 
20 were used for reconstructions with red, green and blue light, respectively (see Section to 
reflect cutoffs below which noise dominates signal. These cutoffs preserved approximately 400 

eigenvectors of HH T for green light, implying a 20 x 20 pixel resolution consistent with ^^p^ . 
Using Equation |2771 , we reconstructed presented images up to this resolution limit as in Figure |9|. 

3. Robustness to Changes in Imaged Wavelength 

Although the prototype PFCA was designed for green light, we also calibrated and tested it with 
blue and red light from the CRT. For each wavelength of light, a separate H was calculated (see 
Equation [21)1) and images were reconstructed as for green light (see Equation P77|) . The recon- 
structed images for all three colours can be found in Figure [ITJ 

As expected, blue light does not produce images of the same quality as green light, and red 
light yields much worse reconstructions. The poor reconstructions available from red light are due 

5 We also attempted ridge regression while determining H in Equation but it did not improve our results. 
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Figure 9. Image Reconstructions. Using the basis functions obtained in the calibration phase (Fig. [7|), we 
reconstructed (B) the image presented (A) up to the Nyquist limit of our array. No off-chip optics were used; 
accumulation times were 16.7 ms. 



Red Light Green Light Blue Light 




Figure 10. Three Colour Image Reconstructions. Using the basis functions H obtained from red, green 
and blue calibration runs, the first prototype chip is able to resolve light images other than at the design 
wavelength (520nm). The center green image is identical to that of Figure 0B on the left. 

to the large gap in Fourier information available in the b = 13-21 range 6 (see Figure H) and the fact 
that red light cannot pass easily through the smallest gratings in use due to its larger wavelength 
resulting in lower signal to noise ratios. 

4. Compressed Sensing Reconstructions 
4.1 Brief Introduction to Compressed Sensing 

Compressed sensing (CS) is a recently-developed signal processing tool useful for reconstructing 
signals with only partial observations IfPl. In general, one can only reconstruct a signal with as 
many (or fewer) free parameters as observations. Partial observations imply a degraded reconstruc- 
tion. However, if the signal is known to be sparse in some basis, then incomplete observations in 
some other basis (unrelated to the basis in which the signal is sparse) can be sufficient to determine 
the signal with high precision. For example, natural scenes are sparse in the space of wavelets iFRIl 

6 The PFCA's blind spot to frequencies in this range is more evident in Figure |l2|, columns Al and A2, rows G 
and B. These images are derived from Equation [Z]7| using the H for red light but with green and blue light incident, 
leading to high noise. Noise in the b = 13-21 range is especially prominent, leading to artifactual "bull's eyes" in the 
reconstruction. 
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(which, roughly speaking, encode edge information) since most scenes contain far fewer edges 
than pixels. Furthermore, since most scenes are well described by their edges, one can encode 
most of the information about the pixels in a scene by encoding the strength of the edges in the 
scene. Thus, a typical natural scene, when mapped onto a wavelet basis set, can be well approx- 
imated by a sparse collection of wavelets with large coefficients. In general, if a system with N 
free parameters (pixels) can be accurately described with k (the number of edges) large-magnitude 
components, and the system is sparse (k <C N), then accurate reconstruction can be performed from 
only M measurements, even if M < N, provided M > k by a comfortable 7 margin. This is true 
because only k free parameters need to be fit. Thus, by enforcing sparsity as well as accuracy, one 
can reconstruct many signals with fewer measurements than there are free parameters. 

Compressed sensing provides a mathematical framework for arriving at a sparse solution that 
still explains observations using convex optimization 8 . CS operates first by assuming that data y 
form a M x 1 vector derived from the N x 1 sparse signal x multiplied byaMxA/' calibration matrix 
A. The goal, then, is to find a signal x that accurately estimates x based on y. To make Ax close to 
y, minimizing the mean square error ||y — Ax||2 is desirable. To enforce sparsity, the L\ norm of x 
||x|| i should also be minimized lOl. Combining these two minimizations with relative importance 
A yields the basis pursuit denoising (BPDN) problem lfI311: 



1 rs 

x = argmin-||y — Ax|| 2 + A||x||i (4.1) 

Solving Equation [O] leads to a sparse explanation of the signal, which will coincide with the true 
signal if it is indeed sparse. 

4.2 Compressed Sensing Permits Full Reconstructions from Incomplete PFCA Information 

The basis in which natural images are sparse is close to a wavelet pyramid IfRll. which is differ- 
ent enough from the Fourier basis that a sparse collection of ASPs from a Fourier pixel should be 
sufficient to reconstruct a full image, provided it is sparse in the wavelet domain. We therefore pro- 
jected H into an overcomplete (2500 x 8619) wavelet basis IfTSIl W and declared A from Equation 
fTTl to be HW. The CS reconstruction problem for recovering the image given limited observations 
becomes 

scs = (W T W)- l W T ^argmn^||r sub -H sub Wx||i + A||x||^ (4.2) 

where su b indicates only a randomly-chosen small subset (40%, 25%, 10% or 8%) of the responses 
r are kept to simulate incomplete measurements. CS reconstructions are of relatively high quality 
despite randomly subsampling Fourier space, as seen in Figure [IT]. 

As part of related work solving underdetermined imaging problems [0L we wrote the in-crowd 
algorithm for solving BPDN lfT5ll. which is faster than all alternative methods on sparse, large-scale 

7 See [Pi for a theoretical guarantee of what "comfortable" means in this context, although provable guarantees place 
much more stringent requirements than practically necessary. 

8 That the problem be posed as convex optimization is important because it eliminates the possibility of local optima, 
meaning the task of searching for a sparse signal compatible with incoming data is tractable. 
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Figure 11. Compressed Sensing Results. We show compressed sensing reconstructions of the test image 
using a randomly-selected subset of ASP sensors. Proportion of sensors whose data are used shown in 
lower-left corner. 

BPDN problems such as this one. On a modern desktop computer, computing the exact BPDN 
solution for our problems takes approximately one second with the in-crowd algorithm, whereas 
other popular alternatives such as GPSR may take longer than an hour with the same inputs. 

PFCAs are thus able to relate complex images to a certain degree even when over 90% of 
their constituent sensors malfunction. Alternatively, it would be possible to design a PFCA with a 
random collection of only 10% of the sensors traditionally considered essential for Fourier coverage 
and still be able to reconstruct images, provided they are sparse in some known, non-Fourier basis. 
This ability is derived from the fact that each ASP makes measurements in a basis in which natural 
images are not sparse (i.e. Fourier components), yet natural images have a known sparsity. There is 
only one sparse combination of wavelets that satisfies the incomplete Fourier measurements taken 
by the PFCA; this is the solution Scs of Equation [0|. 

5. Determining Colour Blindly 

Up until now, we have assumed the reconstruction process has access to information about the 
wavelength of the light imaged. Introducing uncertainty in the wavelength imaged makes recon- 
struction underdetermined by a factor equal to the number of possible wavelengths of light, since 
the number of unknowns becomes the number of pixels times the number of As. However, recover- 
ing full colour information may still be possible. In Section fO| we showed that an undercomplete 
subset of ASP readings can be sufficient to reconstruct a full image. Moreover, as shown in Figure 
0, the b of every device is wavelength-sensitive, meaning that PFCA signals should be sensitive 
enough to X to discriminate colours. Given this wavelength sensitivity along with the robustness to 
undercomplete measurements as shown in Section fO|, it should be possible to determine the colour 
of light incident on the PFCA merely by assuming the image is composed of a sparse combination 
of wavelets of monochromatic light. 

To perform this blind chromatic reconstruction we first computed the matrix of the coupling 
of red, green and blue wavelets to the sensor ensemble: [H re dHg r eenHbi U e]W3 where W3 denotes 
a triple-tall matrix of wavelet coefficients. Next, we found the combination of these wavelets (of 
whatever mixture of colours) that is simultaneously sparse and satisfies the observed ASP outputs 
r by solving Equation |5TT|. 

scs = (WjW 3 )- 1 W 3 r (argmin^||r- ^ (5.1) 

The solution to Equation |5TT| is the triple-tall red, green and blue vector of pixel intensities com- 
posed of a combination of a small number of monochromatic wavelets. The solutions to Equation 
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Figure 12. Blind Colour Imaging. We show reconstructions of test images 1 & 2 without foreknowledge 
of the colour of incident light. R, G and B rows: presented images were red, green and blue. A columns: 
image reconstructed using Equation ^]7| based on H calibrated using red light. B & C columns: as for A, but 
using green- and blue-calibrated H matrices. D columns: reconstruction based on Equation |57T] without any 
assumption about colour, or that the image is monochromatic. 



[577] are plotted in Figure [T2|. This technique does a remarkable job of discovering the correct colour. 
Note that there is no constraint that all wavelets found by Equation |5TT] must be of the same colour 
(and numerical inspection reveals that the images in D columns are not strictly monochromatic); 
merely enforcing sparsity of the solution and consistency with the observations selects wavelets 
that happen to be of the correct colour. Note also that using compressed sensing greatly improves 
the performance of the PFCA for red light: compare R, Dl to R, Al of Figure [T2|. Unexpectedly, 
Fourier information from the missing band b = 13-21 has been filled in. CS gives no indication 
this completion should be possible since it prescribes random sampling (with no systematic, large 
blind spots), yet it performs admirably despite a lack of theoretical guarantee. 
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6. Conclusions 

We have demonstrated techniques for reconstructing images acquired by PFCAs that provide ro- 
bustness beyond what was initially imagined possible. First, we characterized wavelength robust- 
ness of our PFCA. Although designed for green light, our first prototype PFCA performs well under 
blue light using standard linear algebra techniques (Figure [ITJ right) and even under red light us- 
ing basis pursuit denoising and compressed sensing (Figure [T2| R, Dl and R, D2). Reconstruction 
with red incident light is particularly impressive given the large gap in spatial frequencies sampled 
under red light (Figure |5|) and the poor performance of ASPs whose p approaches X. Second, 
we demonstrated robustness against losing all signal from a randomly-chosen set of ASPs. Using 
compressed sensing, we formulated a basis pursuit denoising problem (Equation fOl) that permits 
impressive reconstructions even with a randomly-chosen set with 90% of the signals destroyed 
(Figure [TT|). Last, we applied the same compressive sensing techniques to determine the colour of 
the images presented. The reconstructed wavelengths are nearly flawless (Figure |T2|), and in the 
case of red illumination this technique yields an image of much higher quality than permitted by 
linear algebra reconstructions (Figure [ITJ left). In principle there is no obstacle to taking polychro- 
matic images using a PFCA since Equation |57T] does not require images to be monochromatic, and 
it should be possible as well to recover hyperspectral information (i.e. more than 3 colour chan- 
nels) in a compressed sensing framework, especially using a PFCA designed with a greater number 
of device types. Each device type has an idiosyncratic b(X) with sharp discontinuities caused by 
abruptly rounding effective depth to the nearest optimal Talbot depth (see Figure f| and Equation 
|23l) , providing much better X sensitivity than initially thought. 

In summary, PFCAs enjoy several unusual forms of robustness stemming from their operating 
principles, including insensitivity to lost ASP sensors, flexibility in terms of incident light wave- 
length, and the ability to determine source image colour without employing any chromatic filters. 
These unexpected forms of redundancy enhance the range of applications appropriate for this new 
class of sensing device. 
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